Aligning grid lines of a table in an image of a filled-out paper form with grid lines of a reference table in an image of a template of the filled-out paper form

ABSTRACT

Methods, apparatuses, and embodiments related to aligning an image of a table of a form with an image of a table of a template of the form. Automatically extracting data entered in fields of a table of a form by a user can be aided by matching the form with a template of the form. The form template can have a digitized representation that identifies locations of fields of the form, and that identifies labels of the fields. Matching the form with the form template can enable locations and labels of fields of the form to be identified based on the digitized representation. However, matching the form with the form template may require matching the table of the form with the table of the form template, and matching two tables can be challenging. For example, the tables can be rotated, warped, scaled, etc. relative to each other.

CROSS-REFERENCE TO RELATED APPLICATIONS

This is a non-provisional application filed under 37 C.F.R. § 1.53(b),claiming priority under U.S.C. Section 119(e) to U.S. Provisional PatentApplication Ser. No. 62/257,573 filed Nov. 19, 2015, the entiredisclosure of which is hereby expressly incorporated by reference in itsentirety.

BACKGROUND

Filling out paper forms is a part of life. A trip to a doctor's office,to the department of motor vehicles (DMV), to an office of a potentialnew employer, etc., often involves filling out a paper form. Such formshave fields for people to provide information, such as a field for aperson's name, another for his address, yet another for his phonenumber, etc. The forms can also include a table, and the fields thatpeople fill in can be fields in the table. An employee of the doctor,the DMV, etc. often electronically captures the information entered onthe form by manually entering the information into a computer. Onceelectronically captured, the information can be added to a database, aspreadsheet, an electronic document, etc., where the information can bestored for future reference.

BRIEF DESCRIPTION OF THE DRAWINGS

One or more embodiments are illustrated by way of example in the figuresof the accompanying drawings, in which like references indicate similarelements.

FIGS. 1A-D are flow diagrams that illustrate an example process foridentifying a field on a form template based on an image of the formtemplate, consistent with various embodiments.

FIG. 2 is diagram that illustrates a mapping between an image of a formtemplate and an associated data structure, consistent with variousembodiments.

FIG. 3 is an illustration of a blank school registration form,consistent with various embodiments.

FIGS. 4A and 4B are illustrations of a Department of Motor Vehicles(DMV) form, consistent with various embodiments.

FIG. 5 is an illustration of a blank DMV form, consistent with variousembodiments.

FIG. 6 is an illustration of a pH Indicator table, consistent withvarious embodiments.

FIG. 7 in an illustration of a disease data table, consistent withvarious embodiments.

FIG. 8A is an illustration of a template pH Indicator table, consistentwith various embodiments.

FIG. 8B is an illustration of a first image of a pH Indicator data tablewhere the first image is shrunken relative to the template pH indicatortable, consistent with various embodiments.

FIG. 8C is an illustration of a second image of a pH Indicator datatable where the second image is rotated relative to the template pHindicator table, consistent with various embodiments.

FIG. 8D is an illustration of a third image of a pH Indicator data tablewhere the third instance is warped, consistent with various embodiments.

FIG. 9 is a flow diagram that illustrates a method for aligning gridlines of an image of a table of a form with grid lines of an image of atable of a template of the form, consistent with various embodiments.

FIG. 10 is a flow diagram that illustrates a method for generating athin feature image of a form, consistent with various embodiments.

FIG. 11 is a flow diagram that illustrates a method for generating arotation aligned representation of a table of a form, consistent withvarious embodiments.

FIG. 12 is a flow diagram that illustrates a method for scaling andshifting an image of a form that includes a table, consistent withvarious embodiments.

FIGS. 13A and 13B are flow diagrams that each illustrate a method ofrefining an alignment of an image of a table of a form with an image ofa table of a template of the form, consistent with various embodiments.

FIG. 14 is a block diagram illustrating an example of a processingsystem in which at least some operations described herein can beimplemented, consistent with various embodiments.

FIG. 15 is a graph illustrating a line being expressed with parameters (

,θ), consistent with various embodiments.

FIG. 16 is a graph illustrating a sinusoid for x₀=8 and y₀=6 in a planeθ-

, consistent with various embodiments.

DETAILED DESCRIPTION

Introduced here is technology related to automatically aligning imagesof two tables, such as aligning a table of an image of a form with acorresponding table of an image of a template of the form. A form is apiece of material, such as a piece of paper, plastic, fabric, cardboard,etc., on which data/information/graphics/etc. that defines the form isprinted, written, etc. For example, a form can be a piece of paper whichon which a client intake questionnaire for a doctor's office is printed,can be a piece of paper on which an information data sheet for theDepartment of Motor Vehicles (DMV) is printed, can be a piece of plasticfor an overhead projector on which a teacher draws a table for gatheringstudent preferences for a field trip, can be a cardboard box for acereal on which a contest entry sheet is printed, etc.

The data/information/graphics/etc. that defines a form can be applied inany of various ways to the piece of material of the form, such as bybeing manually written on the piece of material, by being printed on thepiece of material, etc. When the data/information/graphics/etc. of aform is printed on a piece of material, thedata/information/graphics/etc. can be printed by running an applicationprogram, such as a word processor or a spreadsheet program (amongothers), opening a source file which contains thedata/information/graphics/etc., and printing thedata/information/graphics/etc. on the material. For example, a personcan execute a word processor application, can enterdata/information/graphics/etc. that defines a particular questionnaireusing the word processor, and can write thedata/information/graphics/etc. to a source file for the word processor.Anyone else with access to the source file can open the source fileusing the word processor application, and can print thedata/information/graphics/etc. on a piece of paper, turning the piece ofpaper into a form that contains the particular questionnaire. The formcan include graphics, such as boxes, tables, lines, etc., that helpdefine fields where information can be entered by users of the form.

A field is a space on a form where an item of information can be enteredby a person filling out the form, such as by being written or typed inthe field. A field can be in any of various configurations. For example,a field can be a standalone field, can be one of multiple fields, can bea field in an array of fields of a table, etc. A field can containinformation related to identification of the field, such as the name orlabel of the field, among others. For example, a field can includeidentifying information, such as a text string that identifies aparticular field, a text string that identifies a row or column of atable, etc. For example, a field can include a label that identifies allthe fields in a first row of a table as being in row 1. Examples offield labels include a name field, an address field, a driver's licensenumber field, a social security number field, a comment field, a symptomdescription field, a date field, a signature field, a row one column onefield in a table, a row two column five field in a table, etc.

After a person fills out a form, which entails entering data/informationin the various fields, the data/information often needs be to capturedand electronically stored, such as in a database, spreadsheet,electronic document, etc. In many cases, people, such as office staff,read the filled-out forms and manually enter the data/information of theform in a computer, where the information is electronically captured andstored. It is desirable to automate the work that these office workersperform, and to automatically extract and electronically store thedata/information from a filled-out form.

A company, office, or other organization or group may have a number ofdifferent forms. In order to automatically extract information from aparticular form, it may be helpful to identify a form of the multipledifferent forms of which the particular form is an instance. In order toaccomplish such an identification, it can be useful to generate alibrary of templates of the various different forms. If a library ofform templates includes form templates where the fields of the formtemplates have been pre-identified, this pre-identification of thefields can be used to help automate or accelerate extraction andrecognition of the data entered by users in these fields.

A template of a form, also referred to herein as a form template, is aversion of a form that is used as a reference, such as for a comparisonto an image of a selected form to determine whether the selected form isan instance of the form template. A form template can be in any ofvarious forms or formats from which an image of the form template can begenerated. For example, a form template can be a piece of material, suchas a piece of paper, plastic, fabric, cardboard, etc., on whichdata/information/graphics/etc. that defines the form template isprinted, written, etc. In such a case, an image of the form template canbe generated by taking a photo of the form template, by scanning theform template, etc. Additionally, a form template can be a source filefrom which an image of the form template can be generated by use of anapplication program that is compatible with the source file. In such acase, the application program can open the source file, and can generatean image of the form template. In an example, a form template is storedin a source file for a word processor. A user executes the wordprocessor application, accesses the source file, and uses theapplication program to generate an image of the form template, such asby generating a JPEG (Joint Photographic Expert Group) file, TIFF(Tagged Image File Format) file, etc.

A worker, such as an Information Technology person at a doctor's office,can generate a library of images of form templates and can pre-identifyfields of the form templates. The worker can generate images of thevarious form templates in various ways, such as by scanning a first formtemplate when the first form template is a piece of material, by takinga photo of a second form template when the second form template is apiece of material, by running a word processor application to open andgenerate an image of a third form template when the third form templateis a source file for the word processing application, etc. In additionto generating the images of the form templates, the worker can identifyfields of the various form templates. For example, the worker can draw arectangle that identifies the location of a first field of a formtemplate, and can label the first field as the “NAME” field. The workercan similarly draw a rectangle that identifies the location of a secondfield of the form template, and can label the second field as the“ADDRESS” field. The worker can continue this process until heidentifies all of the fields of the various form templates.

When an image of a selected form is received, such as an image of a formcompleted by a user, a computer can compare the image of the selectedform to images of form templates from the form template library. Whenthe image of the selected form matches an image of a particular formtemplate, the selected form can be identified as an instance of theparticular form template. Once the image of the selected form is matchedto the image of the particular form template, the pre-identified fieldsof the particular form template can be used to extract data from thefields of the selected form.

In an example, a computer compares an image of a filled-out DMV formwith various images of form templates from a library of images of formtemplates of the DMV. The computer matches the image of the filled-outform with an image of a form template from the library of DMV formtemplates. The match can be based on a comparison of the entire form, orjust a portion or portions of the form. The various fields of the DMVform template, such as the NAME field and the ADDRESS field, have beenpre-identified and are stored with the library of DMV forms. Thecomputer uses the pre-identified field data of the form template toidentify the location of the NAME and ADDRESS fields in the image of thefilled-out DMV form, extracts the data entered by the user in thosefields, and uses optical character recognition (OCR) to recognize textentered by a user in those fields. The computer further uses thepre-identified labels of the fields to store the extracted text in adatabase. The computer stores the text string from the NAME field in aNAME database entry associated with the filled-out instance of the DMVform, and stores the text string from the ADDRESS field in an ADDRESSdatabase entry associated with the filled-out instance of the DMV form.

In another example, a worker obtains an image of a form template byscanning or taking a photo of the form template. A form template ispreferably a form that is blank (i.e., not filled out by a user), but insome embodiments can also be a version of the form that has been filledout by a user. The worker views the image of the form template using acomputer. The worker uses a mouse to enter a first box that defines thebounds of a first field on the image, uses the mouse to enter a secondbox that defines the bounds of a second field on the image, etc. The“bounds” or “boundary” of a field is one way to define the “location” ofthe field. The worker next provides an identity/label for each box. Forexample, the worker selects the first box using the mouse, and enters“NAME” using a keyboard, thereby identifying/labeling the first box asthe “NAME” field. The worker next selects the second box and enters“PHONE NUMBER”, thereby identifying/labeling the second box as the“PHONE NUMBER” field.

The form template of the example also includes a ten by ten data table.The worker one by one identifies boxes that represent theboundary/location of each of the one hundred fields of the data table,and types in the identity/label of each of the fields. For example, theworker identifies the row one column one field box and types “R1C1” toidentify the field as the row 1 column 1 field. Once all of the fieldsof the form have been located and identified, the worker, using thecomputer, creates a digitized representation of the form template, whichincludes the locations and identities of all of the fields.

Once the digitized representation of the form template has been created,information entered by a user in the various fields of a completed formcan be automatically extracted from an image of the completed version ofthe form. In this example, a computer obtains an image of a completedform that was generated by scanning or taking a photo of the completedform. The computer matches and aligns the image with an image or aportion(s) of an image from a library of form templates. Once the imageis matched with the image or the portion(s) of the image of the formtemplate, the pre-identified locations and identities of the variousfields of the form template can be used to locate corresponding fieldson the completed form. For example, (X,Y) coordinates of the “name”field box, or the “R1C1” field box, can be used to locate an area on thecompleted form where a person's name or the row one column one tabledata is expected to appear.

In order to match an image of a form that includes a table with an imageof a form template that includes a table, the image of the table of theform may need to be aligned with the image of the table of the formtemplate. Aligning two images of a form or a table can be challengingfor a number of reasons. When two images of a form or a table areobtained, the two images can be at a different scale, a differentrotation, etc., relative to each other, or can be offset relative toeach other. Further, either or both of the two images can be warped orotherwise disfigured. For example, an image of a form can be warped orotherwise disfigured when the image is captured by a camera or otherimage acquiring device that was oriented at an angle relative to theform, or if the paper on which the form is printed was bent or otherwisedeformed when the image was captured.

In a first alignment example, a table has regularly spaced horizontaland vertical grid lines, where the grid lines define the table. Whenaligning two instances of such a table, such as a first instance that isan image of a table of a completed form, and a second instance that isan image of the corresponding table of the form template, the firstinstance can be moved relative to the second instance in order to causethe grid lines to align. As the first instance of the table is movedrelative to the second instance in an attempt to align the twoinstances, many of the grid lines can match even though the instancesare not properly aligned.

In an example in which a 10×10 table has eleven vertical grid lines thatare all equally spaced (forming ten fields in a horizontal row of thetable), as a first instance of the table is swept from left to rightacross a second instance, the right-most vertical grid line of the firstinstance first aligns with the left-most vertical grid line of thesecond instance. Then, after the first instance moves one grid linespace further to the right, the two right-most vertical grid lines ofthe first instance align the with two left-most vertical grid lines ofthe second instance, etc. As the first instance continues to move fromleft to right relative to the second instance, more and more of thevertical grid lines align, until finally all eleven vertical grid linesalign. As the sweep continues, the vertical grid lines continue toperiodically align, with one less pair of vertical grid lines aligningeach time the first instance moves right by a grid space. One of thechallenges to properly aligning the two instances of the table is todistinguish between the many incorrect vertical grid alignments betweenthe two tables and the one correct vertical grid alignment.

In a second alignment example, two instances of a form, which include atable, are at different scales. For example, a first instance can be animage of the form at a 90% scale, and a second instance can be an imageof the form at full scale. In general, examples that include twoinstances of a form can also be representative of an example thatincludes an image of a form, and an image of a template of the form. Thesecond instance can be an image of a template of the form. In thisexample, this scaling, or shrinking, results in the X and Y dimensionsor pixel counts of the table of the first instance being 90% of the Xand Y dimensions or pixel counts of the table of the second instance.Another one of the challenges to properly aligning the two instances isto scale the first instance or the second instance to bring the twoinstances back to a same scale. Scaling the first instance or the secondinstance can include scaling both instances.

In a third alignment example, two instances of a form are at differentorientations. For example, a first instance may be an image of the format a first orientation, and a second instance may be an image of theform at a second orientation. The first orientation differs from thesecond orientation by a rotation angle of ten degrees in this example.Rotating the first instance or the second instance to bring the twoinstances back to a same orientation is another one of the challenges toproperly aligning the two instances of the table.

In a fourth alignment example, one or both instances of a form arewarped. In an example, both a first instance and a second instance of aform are warped. When an image is warped, lines that were straight inthe source of the image do not appear as being straight in the image.De-warping the first instance and the second instance is yet another oneof the challenged to properly aligning the two instances of the table.

After two instances of a form are aligned and fields of the twoinstances are correlated, OCR can be run on the areas of the form thatcorrelate to the field boundaries, and the person's name, the row onecolumn one table data, etc. can be automatically extracted based on theOCR results. For example, the boundary of a field can be oversized by apredetermined amount, and OCR can be run on the area of the form thatoverlaps with the oversized boundary box to determine text that appearswithin the oversized boundary box.

Further, when the digitized representation of the form template includesan identity or label of each field, text entered in each field can beautomatically stored in a proper location in a database entry. Forexample, a database entry can be created for a particular completedform. Referring to the example above, a person entered “John Doe” in the“name” field, and “2.0” in the “R1C1” field of this particular completedform. An OCR of the “name” and “R1C1” fields of this particularcompleted form determined that the text in these field is, respectively,“John Doe” and “2.0”. The computer system writes “John Doe” to the“name” field, and “2.0” to the “R1C1” field, of the database entry forthis particular completed form.

In a second example that advantageously utilizes some techniquesdisclosed in this application, a user similarly obtains an image of aform template by scanning or taking a photo of the form template. Theform template is preferably blank, but in some embodiments can also befilled out. The user views the image of the form template using acomputer. The user moves a cursor to a first field of the form template,and the computer automatically displays a predicted location of thefield, including a bounding box that represents the boundary of thefield. The computer further predicts the field identity/label based ontext in the document. The user clicks on the field to indicate that hewants to digitize the field. In some embodiments, the user caninteractively modify the size of the bounding box that represents theextent of the field, and can change the identity/label of the field.Once finalized, the user can cause the field information (e.g., thebounding box coordinate, the bounding box location, the identity/labelof the field, etc.) to be written to a database.

The user now wishes to digitize a table of the form template. The userdepresses a click/select indicator of a mouse outside one corner of thetable and moves the cursor to outside the opposite corner of the table,which causes a table indicator box to be drawn such that the boxencloses the table. The computer system predicts the locations of fieldsof the table, as well as field identities/names/labels for the fields.The user indicates that he wants to digitize the fields of the table. Insome embodiments, the user can interactively modify the size of thebounding boxes that represent the extents of the fields of the table,and can change the identity/label of the fields of the table. Oncefinalized, the user can cause the field information (e.g., the boundingbox coordinates, the bounding box locations, the identity/label of thefields, etc.) for fields of the table to be written to a database.

References in this description to “an embodiment”, “one embodiment”, orthe like, mean that the particular feature, function, structure orcharacteristic being described is included in at least one embodiment ofthe present invention. Occurrences of such phrases in this specificationdo not necessarily all refer to the same embodiment. On the other hand,the embodiments referred to also are not necessarily mutually exclusive.

Further, in this description the term “cause” and variations thereofrefer to either direct causation or indirect causation. For example, acomputer system can “cause” an action by sending a message to a secondcomputer system that commands, requests, or prompts the second computersystem to perform the action. Any number of intermediary devices mayexamine and/or relay the message during this process. In this regard, adevice can “cause” an action even though it may not be known to thedevice whether the action will ultimately be executed.

Note that in this description, any references to sending or transmittinga message, signal, etc. to another device (recipient device) means thatthe message is sent with the intention that its information contentultimately be delivered to the recipient device; hence, such referencesdo not mean that the message must be sent directly to the recipientdevice. That is, unless stated otherwise, there can be one or moreintermediary entities that receive and forward the message/signal,either “as is” or in modified form, prior to its delivery to therecipient device. This clarification also applies to any referencesherein to receiving a message/signal from another device; i.e., directpoint-to-point communication is not required unless stated otherwiseherein.

FIGS. 1A-D are flow diagrams illustrating an example process foridentifying a field of a form template based on an image of the formtemplate, consistent with various embodiments. The process of FIGS. 1A-Dcan be used to identify fields of a form template in preparation, forexample, for generating a library of form templates where the libraryincludes or is associated with digitized representations of the formtemplates that include locations or identities/labels of fields of theform templates. A digitized representation of a form template thatincludes locations or identities/labels of fields of the form templatecan include both locations and identities/labels of the fields of theform template. Identifying a field of a form template can includeidentifying a location of the field on the form template, or identifyingan identity/label of a field of the form template.

At block 110, a computer system receives binary data that represents animage of a form template, such as form template 300 of FIG. 3. Thebinary data can be created in any compatible manner, such as by scanningthe form template, taking a photo of the form template, running anapplication program to access and generate an image of a form template(e.g., when the form template is a source file), etc. The binary datacan be from a scanner, camera, etc. that is coupled to and/or integratedwith the computer system, can be from a remote computer system, can befrom a mobile device such as a smart phone or tablet, can be from anapplication program, etc. The remote computer can have a scanner,camera, etc. that is coupled to and/or integrated with the remotecomputer system, and that can be used to obtain an image of a formtemplate based on a scan or photograph of the form template.

In some embodiments, the image includes meta-data that identifies visualdata, such as meta-data that identifies locations of lines, fields inthe form template, etc. In other embodiments, the image includes nometa-data that identifies visual data, such as locations of lines,locations and/or extents of fields in the form template, etc. In suchembodiments, the process of FIGS. 1A-D advantageously is able to extractvisual data from the image without relying on or needing meta-data thatidentifies or helps to identify the visual data.

At block 130, the computer system creates a data structure to representthe form template. In some embodiments, the data structure is structuredto enable efficient location of fields based on interactive user input.In one example usage scenario, a user views an image of a form templateduring a process of digitizing the form template. When the user moves acursor over a possible location of a field of the form template, thedata structure can be structured to enable a fast and efficientprediction and display of a possible field. Digitizing a form templatecan include storing data related to fields of a form template, such aslocations of fields, boundaries of fields, labels of fields, etc., at adata structure to facilitate automated or semi-automated extraction ofdata written/typed/etc. at the fields of a filled-out version a formthat is an instance of the form template. A label of a field can also bereferred to as an identity of the field.

In some embodiments, the data structure is organized so that adjacentportions of an image map to adjacent elements of a data structure. Forexample, data structure 230 of FIG. 2 is organized so that adjacent“portions” of image 220 map to adjacent elements of data structure 230.Each square of image 220 represents a “portion” of image 220, and eachsquare of data structure 230 represents an element of data structure230. Each corner of each square of image 220, such as the square atindex (0,0), or the square at index (8,8), is coincident with a grid ofimage 220. Grid points 225 identifies examples of three grids, alsoreferred to as grid points.

As can be seen at mapping 210 and mapping 215, two adjacent “portions”of image 220 (e.g., portion (0,0) and portion (1,0)) map to adjacentelements of data structure 230 (e.g., element (0,0) and element (1,0)).

Blocks 132-138 represent a flow diagram to perform the operation ofblock 130. At block 132, the computer system analyzes the image of theform template. Features of a data structure can be determined by acomputer system based on the analysis of the image. For example,features of a data structure, such as the number of elements of the datastructure, the organization of the data structure, the data associatedwith each element, etc. can be determined based on an analysis of theimage.

In some embodiments, the computer system determines a data structurebased on a grid that the computer system determines based on the imageof the form template. A grid can define the “portions” of an image. InFIG. 2 for example, grid points of image 220, such as grid points 225,define “portions” of image 220, where each non-overlapping square ofimage 220 represents one of the “portions” of image 220.

The computer system can determine the size of a grid based on ananalysis of the image. For example, a grid can be set based on the sizeof a pixel of the image (e.g., grid=1 pixel×1 pixel), the size of agrouping of pixels of the image (e.g., grid=3×3 pixels), a measuredportion of the image (e.g., grid=0.1 mm×0.1 mm), a percentage of a sizeof the image (e.g., the X grid=1% of the X dimension of the image, andthe Y grid=1% of the Y dimension of the image), etc. When the grid is apixel, one of the “portions” of image 220 includes data of the pixelthat overlaps with one of the squares of image 220. When the grid isgroup of 3×3 pixels, one of the “portions” of image 220 includes data ofthe 9 pixels that overlap with one of the squares of image 220. Etc.

While the current discussion focuses on a regular grid whose grid pointsdefine squares, a grid can have various characteristics that aredetermined based on any of various other data. For example, a grid candefine a set of non-overlapping rectangles, such as when the grid is 2pixels by 1 pixel. As another example, the grid can be irregular. Forexample, the grid of FIG. 7 can be coincident with the lineintersections of FIG. 7, where the lines and associated lineintersections are irregularly spaced.

The computer system can determine a data structure (block 134) based onresults of the analysis of block 132. In some embodiments, the computersystem determines the data structure to enable data of each “portion” ofthe image to map to a different element of the data structure, and toenable adjacent “portions” of the image map to adjacent elements of thedata structure. Data structure 230 of FIG. 2 is an example of such adata structure.

The computer system can determine any of various types of datastructures, such as a linked list, an array, a hash table, etc. Further,the data structure can be based on any of various attributes of theimage, such as the color(s) of the image, the size of the image, theresolution of the image, etc.

In some embodiments, two elements of a data structure are considered tobe adjacent when the index of each of the elements differs by one unit.For example, in data structure 230, the index is an ordered pair. Fordata structure 230, two elements are considered adjacent when one numberof the ordered pair differs by one unit, and the other number of theordered pair is the same. For example, element (0,0) is adjacent toelement (1,0) because one number of the ordered pair (the first numberin this example) differs by one unit, and the other number of theordered pair is the same. Similarly, element (0,0) is adjacent toelement (0,1) because one number of the ordered pair (the second numberin this example) differs by one unit, and the other number of theordered pair is the same.

As discussed above, the data structure can be organized so that therelative locations of the “portions” of the image are reflected in theorganization of the data structure. When a document, such as a formtemplate, is scanned, photographed, etc., the resulting image has acertain resolution. For example, the resolution of the image can be 2048pixels by 1536 pixels, for a total of 3,145,728 pixels (or 3.1Megapixels). In some embodiments, the computer system determines thegrid size so that the grid matches the size of a pixel. In such anembodiment, each pixel of the image is associated with a particular rowand column of the 2048 pixel×1536 pixel image of the form template. Inthis example, a pixel located at numbered pair (0,0) is located at thebottom-left of the image, and a pixel located at (2047,1535) is locatedat the top-right of the photo. As is shown in FIG. 2, adjacent portions(0,0) and (1,0) of image 220 map to adjacent elements (0,0) and (1,0) ofdata structure 230. When the portion is a pixel, adjacent pixels (0,0)and (1,0) of the image map to adjacent elements (0,0) and (1,0) of thedata structure.

At block 136, the computer system identifies a line segment. Somealgorithms for predicting fields in a form template use locations oflines on the form template to predict or determine a location of afield, and a boundary of the field. The location of the field is wherethe field is located on the image of the form template. The location canbe any location associated with the field, such as the location of themiddle of the field, the bottom left corner of the field, the locationof a line that defines a boundary of the field, such as a bottom line ofa field, etc. The boundary of the field defines an area on the imagewhere a user is expected to input (e.g., write, type, etc.) a value forthe field.

Some embodiments of the process of FIG. 1 are interactive, in that userinput is required during the process. In preparation for an interactivesession, a computer system can pre-populate a data structure with dataderived from an image to be digitized to enable a faster and moreefficient interactive user experience. Some embodiments advantageouslyload data of the form template in memory in preparation for aninteractive session. Loading the data of the form template in memory,such as by loading a data structure that contains data of the formtemplate, and organizing the data to efficiently support an interactivesession, can enable a much faster and responsive interactive userexperience.

Some embodiments of the process of FIG. 1 use locations of line segmentsof the form template in predicting or determining field locations andboundaries. In order to facilitate an improved interactive userexperience, qualifying line segments can be stored in a data structurethat is organized/structured to efficiently support an interactivesession. Returning to block 126, in some embodiments, the computersystem identifies all visible line segments of a form template that meeta certain criteria. The computer system does this by analyzing thebinary data of the image of the form template to identify line segmentsthat are visible on the form template. Identifying line segments is wellknown in the art, with one example of an algorithm for doing so being aHough Line Transform, which can be found athttp://docs.opencv.org/doc/tutorials/imgproc/imgtrans/hough_lines/hough_lines.html.A second example of a line extractor can be found athttp://docs.opencv.org/master/db/d73/classcv_1_1LineSegmentDetector.html.

As line segments are found, they can be evaluated to determine if theymeet certain criteria. For example, lines that do not extend apre-determined distance may be filtered out, lines that do not runsufficiently or substantially straight in the X or Y dimension may befiltered out, lines that are or are not of a certain color or colorrange may be filtered out, lines that are or are not of a certain style(such as a dashed line) may be filtered out, etc. In some embodiments, aline is deemed to run sufficiently straight when its length in theprimary direction in which the line runs is at least 600% of the lengththat it traverses in the non-primary direction. For example, a line thatextends 3 inches in the X dimension may be filtered out if thedifference between the maximum and minimum Y coordinates of the linesegment is more than 0.5 inches.

In some embodiments, once a line segment is determined and passes allline segment filters, the line segment is mapped to a data structureelement (block 138). As discussed above, data of a “portion” of an imagethat is determined based on a grid can be mapped to an element of a datastructure. Once the line segment is determined, the computer systemdetermines which “portions” of the image the line passes through, andmaps the line to the elements that are mapped to those “portions.” Forexample, where the grid is a pixel and a line runs through 1000 pixels,the line can be mapped to each of the 1000 pixels. The data stored ateach element can include, e.g., a link to a data structure element thatincludes the data of the line segment, information regarding the portionof the line segment that passes through the portion of the image thatmaps to the element, etc.

In some embodiments, this process is repeated for all line segments ofthe form template, such that, once completed, the data structureincludes line segment information for all line segments that met thecriteria and were not filtered out. Once the process is complete, thedata structure of such embodiments includes information regarding allthe line segments that meet the criteria, thereby advantageouslyenabling an enhanced interactive user experience.

At block 150, the computer system predicts a location of a field of theform template. Any computing device, such as a remote computer system, amobile device, etc. can be used to perform the operation of block 150. Amobile device can be any mobile computing device, such as a laptopcomputer, a smart phone, a tablet, etc. In some embodiments, theoperations of all blocks of FIG. 1 are performed by a computer system.In other embodiments, the operations of blocks 110-138 are performed bya server, and the operations of blocks 150-195 are performed by acomputing device, such as a laptop computer, a desktop computer, a smartphone, a tablet, etc. In other embodiments, various computers performthe operations of the various blocks of FIG. 1A-D.

Blocks 152-158 represent a flow diagram to perform the operation ofblock 150. At block 152, a computer system determines a location of acursor. In some embodiments, when a user begins an interactive sessionto digitize a form template, the user views an image of the formtemplate. The image of the form template can be displayed by any ofvarious applications that can display images, such as an image viewer, aphoto viewer, a PDF viewer, a web browser, a word processor, etc. Theprocess of digitizing the form template includes identifying relevantfields of the form template that a user may want to automaticallyextract from a filled-out form that is an instance of the form template.The user guides a cursor over a field, in this example field 305 of FIG.3.

When the cursor hovers over and/or stays substantially at a selectedlocation on the image, the computer system determines the location ofthe cursor, such as the location of the cursor on the image. Based onthe location of the cursor, the computer system determines a “portion”of the image (block 154). The “portion” of the image can be a portion ofthe image that is located at the location of the cursor. For example,when the grid of image 300 of FIG. 3 is a pixel and the user hovers acursor over field 305, the computer system determines the coordinate ofthe cursor. Based on the coordinate of the cursor, and based on the sizeof the grid (in this example, a pixel), the computer system determinesthat the coordinate of the cursor intersects a particular pixel of theimage (i.e., the pixel being the “portion” of the image). In variousembodiments, the portion of the image that is determined at block 154can be a grid, multiple grids, part of the image that is within adefined distance the location of the cursor, etc.

At block 156, the computer system determines a location of a field basedon a location of nearby line segments, and at block 158 determines anextent of the field. In a first example, a field is encircled by linesegments. An example of such a field is field 405 of FIG. 4A, where auser would enter the “MAKE” of an automobile on a DMV form. As discussedabove, the location of the field can be any location associated with thefield, such as the location of the middle of the field, the bottom leftcorner of the field, the location of a line that defines a boundary ofthe field, the location of an extent of the field, etc.

In this example, the location of the field is the location of the linesegment at the bottom of the field. Further, the data structure of block130 is organized so that adjacent portions of an image map to adjacentelements of a data structure to enable a fast and efficient predictionand display of a possible field. In other words, when a first “portion”of the image is located “below” a second “portion” of the image, theelement that maps to the first portion will be “below” the element thatmaps to the second portion. For example, in FIG. 2, the portion of image220 at index (0,0) is “below” the portion of image 220 at index (0,1).Accordingly, the element of data structure 230 at index (0,0) is also“below” the element at index (0,1).

Based on the cursor location of block 152, and the associated “portion”of the image, the computer system accesses a data structure elementassociated with the “portion” of the image to determine if the elementis mapped to a line segment. In this example, when the element is mappedto a line segment, the location of the line segment is determined andthe location of the field is based on the location of this nearby linesegment (block 156). The location of the field can be based on or can beany location characteristic of the line segment, such as a coordinate ofa predetermined point on the line segment (e.g., the middle, left end,right end, etc. of the line segment), a location of the line segment, alocation of the “portion” of the image that maps to the line segment,etc.

When the element is not mapped to a line segment, the computer systembegins walking “down” elements of the data structure to find a linesegment that is “below” the element. In this example, the index of anelement of the data structure has the format index (X,Y). The computersystem can walk “down” elements of the structure by decrementing Y. Forexample, if the starting element has an index of (1000,900), the nextelement when walking “down” elements of the data structure will have anindex of (1000,899). As the computer system walks down elements of thedata structure, it determines whether the next lower element maps to aline segment. If not, the computer system continues to the next lowerelement. Once the computer system identifies an element that maps to aline segment, the computer system in this example determines thelocation of the line segment and bases the location of the field on thelocation of this nearby line segment (block 156).

Once the “lower” line segment is identified, the computer systemdetermines an extent of the field (block 158). In this first example,the computer system walks “up” elements of the data structure until itfinds a line segment “above” the lower line segment. The computer systemthen walks “left” and “right” until it finds line segments that are“left” and “right” of the starting element. When the four line segmentsform a box, the extent of the field is based on the four line segments.For example, the extent of the field can be defined as the box formed bythe four line segments, or by taking the box formed by the four linesegments and over sizing or under sizing the box to determine the extentand/or boundary of the field.

In a second example, a field is bounded on two or three sides by linesegments. Examples of such fields include fields 410 and 415 of FIG. 4A.In such examples, the location of nearby line segments and the locationof the field are determined in a manner similar to the first example(block 156). However, when searching in a direction where the field isnot bounded by a line, no line is found. When the computer system is inthe process of determining the extent of the field (block 158), eachdiscovered bounding line determines an extent of the field in onedimension. The extent of the field in directions where no bounding linewas discovered is determined based on the end points of the boundinglines. For fields 410 and 415, the left extent is set to be coincidentto the left ends of the top and bottom bounding line segments. For field415, the right extent is similarly set to be coincident to the rightends of the top and bottom bounding line segments.

In a third example, a field is bounded on one side by a line segment. Anexample of such a field is field 305 of FIG. 3. In such an example, thelocation of nearby line segments and the location of the field aredetermined in a manner similar to the first example (block 156).However, when searching in a direction where the field is not bounded bya line, no line is found in that direction. When determining the extentof such a field (block 156), a computer system can set the left andright ends of the field to be coincident with the left and right ends ofthe discovered bounding line segment.

The top of the extent of the field can be determined in any of severalways. For example, the height can be determined based on the height oftext that is associated with the field. For field 305, the top extent ofthe field can be set based on the text to the left of the field,“Child's Name,” which is the text associated with field 305. The heightof the field can be set equal to the height of the associated text, canbe set to be a multiple of the height of the associated text (e.g., 1.5times the height of the associated text), etc. As another example, theheight can be determined based on the average or median height of textin the document. For example, the height of the field can be set equalto the median or average height of text in the document, can be set tobe a multiple of the median or average height of the text in thedocument (e.g., 1.5 times the height of the median or average textheight), etc.

In a forth example, a field includes extraneous line segments. Anexample of such a field is field 505 of FIG. 5. Field 505 includes linesegments to denote where each character should be placed. In such acase, a detected nearby line segment can be filtered based on certaincriteria. The filtering can happen prior to block 156. For example, whenthe line segments that form the top and bottom extents of field 505 arelocated, the Y dimension distance between the two line segments can bedetermined. When searching for line segments in the left or rightdirection, any detected vertical line segments that have a length lessthan a predetermined portion of the distance between the top and bottomextent line segments, such as 25% or 50% of the distance, can befiltered. Once the extraneous line segments are filtered, field 505 isdetermined in a manner similar to field 415.

In a fifth example, a field is bounded on one side by a line segment,and on an opposite side by one or more line segments whose lengths aredifferent than or are offset from the boundary line segment of the firstside. Examples of such fields include fields 310 and 315 of FIG. 3. Insuch cases, where one or more line segments are found in the verticaldirection (block 156) which do not match the length and/or do not lineup with the bottom line segment (e.g., the upper line segment is thesame length, but is shifted in the left or right directions as comparedto the bottom line segment), any of various techniques can be used todetermine the upper extent of the field (block 158). For example, theupper extent of the field can be determined in a manner similar to thethird example where the extent is based on the height of text in thedocument. As another example, the upper extent of the field can be setto be coincident with the line segment(s) found in the verticaldirection, or to be coincident with the lowest line segment found in thevertical direction. When determining whether two line segments have thesame length, or are shifted, the comparison between the line segmentscan include an error threshold, which can be a predetermined absoluteamount, can be a predetermined relative amount, etc. For example, theerror threshold can be 0.1 inches, can be 10% of the length of the linesegment, etc.

In a sixth example, multiple fields are bounded on the top and bottom byline segments that extend the length of the multiple fields, and are notbounded on the left or right. An example of such a field is field 420 ofFIG. 4A, which is bounded on the top and bottom by line segments thatextend past the fields for APPLICANT'S SIGNATURE, for PRINTED NAME, andfor DATE. In such a case, a computer system can determine the left andright extent of each field based on text associated with the fields orby the ends of the top and bottom boundary line segments (block 158).For example, a computer system can determine that multiple text fieldsare associated with the top and bottom boundary line segments based onspacing between the text. A computer system can evaluate the spacebetween words, and when space between some words associated with a linesegment or field exceeds a predetermined threshold, the computer systemcan determine that the large spacing indicates a new field. Thethreshold can be an absolute amount (e.g., a spacing more than 0.25inches in the original image or when printed), can be a relative amount(e.g., a spacing more than two times the height of the text, more than 6times the median spacing between words, etc.), among other amounts. Linesegments 425-440 of FIG. 4B are examples of line segments that representthe left and right extents of the multiple fields of field 420.

In a seventh example, a field is part of a table. Examples of suchfields include fields 605 and 620 of table 600 of FIG. 6, and field 705of table 700 of FIG. 7. In such cases, a table can be automatically orsemi-automatically detected. When tables 600 or 700 are automaticallydetected, a user can place a cursor over a field of the table, such asfield 605 or field 705, and a computer system can determine the locationof the cursor (block 152). The computer system can determine a portionof the image based on the cursor location (block 154) in a mannersimilar to some of the above examples. When determining a field based onlocations of nearby line segments (block 156), which can be done in amanner similar to the first example where the field is encircled by linesegments, a computer system can continue to search in the upper, lower,right, and left directions and can determine, based on the location ofdetected line segments, that field 605 is part of table 600, or thatfield 705 is part of table 700. For example, when the computer systemsearched in the right direction, the computer system can detect a seriesof similarly sized line segments. By doing similar searches in the left,upper, and lower directions the computer system can detect other linesegments of the table in a similar manner, and, based on the detectedline segments, can determine a location and extent of the table.

When tables 600 or 700 are semi-automatically or interactively detected,a computer system can receive input from a user that indicates anexistence and location of the table. For example, when determining alocation of a cursor (block 152), a computer system can also determinethat a user drew a box with a mouse or other device to indicate a table.For table 600/700, a user can click slightly below and to the left ofthe bottom left corner of table 600/700 (e.g., below and to the left offield 615/715), can continue to depress the click indicator while movingthe cursor to slightly above and to the right of the top right corner oftable 600/700 (e.g., above and to the right of field 610/710), where hereleases the click indicator, thereby indicating a first and a secondcorner of a rectangle that indicates an extent of a table, in thisexample table 600/700. The computer can analyze line segments that arewithin the indicated drawn bounding rectangle to determine both thelocation and extent of table 600/700, as well as the location and extentof each of the fields of the table, such as field 605/705 (blocks 156and 158). In table 600/700, each field is encircled by line segments,and the location and boundary of field 605/705 can be determined in amanner similar to the first example where a field is encircled by linesegments.

In some embodiments, X and Y dimensions of table 600/700 are determined,and lines that run less than a predetermined amount, such as 75% of atable dimension, are filtered. For example, if a line runs in the Xdimension less than 75% of the X dimension of the table, or if a lineruns in the Y dimension less than 75% of the Y dimension of the table,the line can be filtered out.

At block 170, a computer system predicts a label for the field. Blocks172-176 represent a flow diagram for performing the operation of block170. A computer system can select a field based on a location of acursor on the image of the form template (block 172). The location ofthe cursor at block 172 can be the same location of the cursor at block152, or can be a different location, for example, due to the cursormoving slightly. The field can be the field of block 156. The computersystem can run OCR on part or all of the image to determine text of theimage (block 174). OCR can be run on the contents of a field when a userindicates a selection of the field, can be run on the entire documentduring creation of a data structure at block 130, or can be run at anyof various other times. The contents of the field can be defined by theextent of the field as determined at block 158, by under sizing or oversizing the extent of the field of block 158, e.g., over sizing theextent of the field by 50% of the height of the field, or by anothermethod.

A label for a field can be predicted in any of various ways (block 176).For example, the label can be predicted based on text that is locatednearby to or at the field. As an example, prediction of labels forfields 305 and 310 of FIG. 3 can be based on an analysis of text/fieldpatterns in a row of document 300 that includes the field. A computersystem can determine that the text CHILD'S NAME is followed by field305, which is followed by CHILD'S SOCIAL SECURITY #, which is followedby another field. The computer system, based on this pattern oftext/field/text/field, can predict that the first text (i.e., CHILD'SNAME) is a label for the first field (i.e., field 305), and that thesecond text is a label for the second field.

As another example, a prediction of the label for field 315 can besimilarly based on an analysis of text/field patterns in a row ofdocument 300. A computer system can determine that the text PARENTS ARE:is followed by a first field, which is followed by MARRIED, which isfollowed by a second field, which is followed by SINGLE, which isfollowed by a third field, which is followed by SEPARATED, which isfollowed by a forth field, which is followed by DIVORCED. In thisexample, with there being text to both the left and right of each field,the computer system can additionally base a prediction of a label for afield on text punctuation. In this case, based on the first text endingin a colon (i.e., “:”), the computer system can predict that the textthat follows each field is the label for the field, and that the labelfor field 315 is DIVORCED.

As yet another example, a prediction of the label for field 405 of FIG.4A can be based on text that is located within the bounds of the field.A computer system can determine that the text MAKE lies within theboundary of field 405, and can predict that MAKE is the label for field405. The bounds/boundary of the field can be defined by the extent ofthe field as determined at block 158, by a sizing of the extent of thefield of block 158, or by another method. As yet another example, aprediction of a label for field 420 of FIG. 4A can be based on text thatis located within the bounds of the multiple fields of field 420, as isrepresented by line segments 425-440 of FIG. 4B. In this example, alabel for the first of the multiple fields of field 420 is APPLICANT'SSIGNATURE, a label for the second of the multiple fields is PRINTEDNAME, and a label for the third of the multiple fields is DATE.

A label for a field in a table, such as fields 605 or 620 of table 600of FIG. 6, or field 705 of table 700 of FIG. 7, can be based on text inthe outermost row(s) and column(s) of the table. In the example of FIG.6, area 625, which includes the text SAMPLES, and area 635, whichincludes the text PH INDICATORS, are not part of table 600. The linesegments that define the extents of areas 625 and 635 and do not definethe extent of table 600 (i.e., the “625/635 line segments”), are notpart of table 600 in this example and can be filtered in any of severalways. For example, when a user draws a box using a mouse or other deviceto indicate a table, the user can draw the box so that it does notinclude the entirety of areas 625 and 635. A computer system can filterthe 625/635 line segments based on their not being fully containedwithin the drawn box.

In another example, a user draws the box so that it includes all ofareas 625 and 635. In this example, a computer system can filter the625/635 line segments based on the Y-dimension of area 625, and theX-dimension of area 635, not matching the X and Y dimensions of fieldsof table 600. In other words, the computer system can analyze fieldsthat it finds within the drawn table indication box. When the computersystem finds a number of abutting fields that are laid out in a tableconfiguration, it can keep those fields, and it can filter out fieldsthat do not match the table pattern. In this example, the 625/635 linesegments that define boundaries of area 625 will be filtered out due tothe Y dimension of area 625 not matching the Y dimension of fields oftable 600. Additionally, the 625/635 line segments that defineboundaries of area 635 will be filtered out due to the X dimension ofarea 635 not matching the X dimension of fields of table 600. Thisfiltering will leave line segments that form the boundaries of thefields that form the rows and columns of the table.

Contents of fields that are in the outermost row(s) and column(s) can beanalyzed to predict labels for the fields of table 600 (block 176). Acomputer system analyzes table 600 and determines that fields of theleft most column and top most row of table 600 include text. Thecomputer system can base the labels of the fields of table 600 on thetext found in these fields. For example, a label of field 605 can be setto “PH METER” “SODIUM CARBONATE NA2CO3”, with “PH METER” being predictedas part of the label for all fields in column 630 that are below the topmost row, and with “SODIUM CARBONATE NA2CO3” being predicted as part ofthe label for all fields of row 640 to the right of the left mostcolumn. As a second example, a label of field 620 can be “RED CABBAGEEXTRACT” “MILK OF MAGNESIA MG(OH)2”.

Labels for fields of table 700 are handled in a manner similar to thefields of table 600. In the example of FIG. 7, a label of field 705 canbe set to “NEW CASES” “MALARIA”, with “NEW CASES” being predicted aspart of the label for all fields in column 720 below the top-most row oftable 700, and with “SODIUM CARBONATE NA2CO3” being predicted as part ofthe label for all fields in row 725 to the right of the left most columnof table 700.

At block 190, the computer system displays a boundary that representsthe field. The boundary can be the extent determined at block 158 ofFIG. 1C, can be the extent oversized or undersized by a predeterminedamount, etc. For example, the boundary can be determined by oversizingthe extent of the field by 0.25 inches, by undersizing the extent of thefield by 10% of the height of the extent, etc. The boundary can bedisplayed in response to the user placing the cursor at a location ofthe field, by the user hovering the cursor or keeping the cursorsubstantially still over the field, etc.

At block 195 the user digitizes the form template. Digitizing a formtemplate can include, for example, storing data related to fields of aform template, such as locations of fields, boundaries of fields, labelsof fields, etc., at a data structure, such as to facilitate automated orsemi-automated extraction of data written/typed/etc. at the fields of afilled-out version of a form that is an instance of the form template.The data structure can be the data structure of block 130, or can beanother data structure. For example, the computer system can create adata structure element for a field, such as for field 305 of FIG. 3. Thedata structure element can include members. For example, as part ofdigitizing the form template, the computer system can create members ofthe data structure element. For example, the computer system can createa member that defines the boundary of the field, another member thatdefines the field label, another memory that stores the location of thefield, etc. The data structure of block 195 can be stored to disk forfuture use, such as when a filled-out version a form that matches theform template is received and the filled-out values of the variousfields are extracted from the form and added to a database.

FIG. 8A is an illustration of a template pH Indicator table, consistentwith various embodiments. Table 600 of FIG. 6 can be a template of a pHIndicator table, and table 800 a can be table 600. Further, fields 805a-820 a, area 825 a, column 830 a, area 835 a, and row 840 a can be,respectively, fields 605-620, area 625, column 630, area 635, row 640 ofFIG. 6.

FIG. 8B is an illustration of a first image of a pH Indicator data tablewhere the first image is shrunken relative to the template pH indicatortable, consistent with various embodiments. Table 800 b is anillustration of an image of table 800 a that is shrunken with referenceto table 800 a. Fields 805 b-820 b, area 825 b, column 830 b, area 835b, and row 840 b of FIG. 8b can be, respectively, fields 805 a-820 a,area 825 a, column 830 a, area 835 a, and row 840 a of FIG. 8a , howeverin an image of table 800 a that is shrunken.

FIG. 8C is an illustration of a second image of a pH Indicator datatable where the second image is rotated relative to the template pHindicator table, consistent with various embodiments. Table 800 c is anillustration of an image of table 800 a that is rotated with referenceto table 800 a. Fields 805 c-820 c, area 825 c, column 830 c, area 835c, and row 840 c of FIG. 8c can be, respectively, fields 805 a-820 a,area 825 a, column 830 a, area 835 a, and row 840 a of FIG. 8a , howeverin an image of table 800 a that is rotated.

FIG. 8D is an illustration of a third image of a pH Indicator data tablewhere the third instance is warped, consistent with various embodiments.Table 800 d is an illustration of an image of table 800 a that is warpedwith reference to table 800 a. Fields 805 d-820 d, area 825 d, column830 d, area 835 d, and row 840 d of FIG. 8d can be, respectively, fields805 a-820 a, area 825 a, column 830 a, area 835 a, and row 840 a of FIG.8a , however in an image of table 800 a that is warped.

FIG. 9 is a flow diagram that illustrates a method for aligning gridlines of an image of a table of a form with grid lines of an image of atable of a template of the form, consistent with various embodiments.While the method of the example of FIG. 9 is used to align grid lines ofan image of a table of a form with grid lines of an image of a table ofa template of the form, the method can be used to align grid lines oftwo images of a table, or grid lines of an image of a first table andgrid lines of an image of a second table where the two tables aresubstantially identical, among others. While, in the example method ofFIG. 9, some actions of the method are being applied to the image of theform, or are being applied to an image that is derived from an image ofthe form, these actions can also be applied to the image of the formtemplate or to an image that is derived from the image of the formtemplate. For example, both the image of the form and the image of theform template may need to be straightened, scaled, shifted, etc. tofacilitate aligning the grid lines of the image of the table of the formwith the grid lines of the image of the table of the form template, tofacilitate matching the image of the form with the image of the formtemplate, etc.

As discussed above, a process for automatically extracting, recognizing,and storing data entered by a user in a field of a selected form caninclude matching the selected form with a form template of a library ofform templates. If the image of the selected form matches an image of aparticular form template, the selected form can be identified as aninstance of the particular form template. Once the image of the selectedform is matched to the image of the particular form template, thepre-identified fields of the particular form template can be used toextract data from the corresponding fields of the selected form. Inorder to match an image of a form that includes a table with an image ofa form template that includes a table, the image of the table of theform may need to be aligned with the image of the table of the formtemplate. Aligning two images of a table can be challenging for a numberof reasons, such as those discussed above.

At block 905, a computer system, such as processing system 1400 of FIG.14, receives an image of a form, and receives an image of a template ofthe form, the template of the form also being referred to as the formtemplate. The images can be received at the same time, or at differenttimes. The image of the form can be generated by use of any imageacquiring device, such as by taking a photo of the form, by scanning theform, etc. The form can be a blank form, can be a filled-out version ofthe form, etc. The image of the template of the form, or the formtemplate, can similarly be generated. Further, the image of the formtemplate can additionally be generated by use of an application program,such as a word processor, a spreadsheet program, etc. The applicationprogram can access an electronic file in which a representation of theform template is stored, and can generate the image of the formtemplate, such as by generating a JPEG (Joint Photographic ExpertGroup), TIFF (Tagged Image File Format), GIF (Graphics InterchangeFormat), PNG (Portable Network Graphics), PDF (Portable GraphicsFormat), etc. The image of the form template can be received from avariety of sources, such as from a camera, from a scanner, from anapplication program, from a remote computer system, etc.

The form template or the image of the form template can also be obtainedfrom a library of form templates. For example, the processes describedabove can be used to generate a library of form templates, wherelocations of all or some of the fields of each form template areidentified and stored with the library of form templates, such as indigitized representations of the form templates. An image of a form canbe matched with a form template from the library of form templates, suchas by matching the image of the form with an image of the form template.A portion of the image defined by a field of the form, such as a portionof the form that is within an oversized version of a rectangle thatdefines an extent of the field, can be automatically extracted from theimage of the form. Further, text or other graphic representations in theportion of the image can be automatically recognized and electronicallystored. This can be repeated for each of the fields of the form.

However, when a form includes a table, it may be necessary or mayincrease compute efficiency to align the image of the table with animage of a table of a form template before or as part of determiningwhether the form matched with the form template. Aligning two imageswhich include a table, such as two images of a table, or an image of afirst table and an image of a second table where the first and secondtables are identical or substantially identical, can be challenging fora number of reasons. For example, the two images can be at a differentscale relative to each other, as is demonstrated in FIG. 8B where table800 b is shrunken with respect to table 800 a of FIG. 8a . As anotherexample, the two images can be at different rotations relative to eachother, as is demonstrated in FIG. 8c where table 800 c is at a differentrotational orientation than is table 800 a. As yet another example,either or both of the images can be warped or otherwise disfigured, asis depicted in FIG. 8d . Table 800 d of FIG. 8d is an image of table 800a, where the image is warped with reference to table 800 a. Table 800 dmay be warped because the image was captured by a camera that wasoriented at an angle relative to table 800 a, because the material onwhich table 800 a is printed was bent when the image was captured, etc.

At block 910, the computer system generates a thin feature image of theform. A thin feature image of a form is an image where wider features ofthe form have been filtered out, leaving thinner features of the form,such as grid lines and text. A thin feature image can further have stepedges filtered out. Step edges are edges formed by a transition betweenconstant regions. An example of a step edge can be when a scan or photoof a form extends beyond the edge of a piece of paper on which the formis printed. The paper side of the edge of the paper can be a first colorin the image, and the other side of the edge of the paper can be asecond color. The step edge of this example is the edge formed by thetransition from the first color to the second color. The thin featureimage can be generated by any of various techniques. For example, afilter algorithm that filters thin features of images can be executed bythe computer system on the image of the form to create a filtered imageof the form that has thin features filtered out. The filtered image ofthe form can be subtracted from the image of the form to generate a thinfeature image of the form. FIG. 10 describes an example of a techniquethat can be used to generate a thin feature image of a form.

At block 915, the computer system straightens grid lines of a table ofthe thin feature image to create a rotation aligned version of the thinfeature image. In some embodiments, straightening the grid lines has twoaspects, a first aspect being to orient the grid lines to achieve adesired orientation with reference to an X or Y axis, and a secondaspect being to linearize the grid lines to cause the grid lines to runin a straight line. The X axis can run horizontally and the Y axisvertically with reference to the thin feature image, or with referenceto alignment of pixels of the thin feature image, and the X and Y axescan be perpendicular. For example, when pixels of the thin feature imageare organized in rows and columns, the X axis can run parallel to therows and the Y axis can run parallel to the columns.

Orienting the grid lines to achieve the desired orientation withreference to the X or Y axis can involve transforming the thin featureimage to achieve a rotation of the thin feature image that has thedesired orientation, such as by applying a first transformation to thethin feature image. For example, when the desired orientation of thegrid lines is to be substantially parallel with or perpendicular to aselected one of the X or the Y axis, the thin feature image can betransformed to achieve a rotation of the thin feature image where thegrid lines achieve the desired orientation.

Linearizing the grid lines to cause the grid lines to run in a straightline can involve removing or reducing bends, warps, discontinuities, orother non-straight features of the grid lines, such as by applying asecond transformation to the thin feature image. The computer system cancreate a rotation aligned version of the thin feature image by applyingthe first transformation and the second transformation, or by applyingany other technique that straightens the grid lines of the table of thethin feature image. In some embodiments, the first transformation andthe second transformation are a same transformation that transforms thethin feature image and both orients the grid lines to achieve a desiredorientation with reference to an X or a Y axis, and linearizes the gridlines. FIG. 11 describes an example of a technique that can be used tostraighten grid lines of a table of a thin feature image to create arotation aligned version of the thin feature image.

At block 920, the computer system scales and shifts the rotation alignedversion of the thin feature image. When an image of a form is created,the image can have been created at a different scale, or at an offsetrelative to an image of the form template of which the form is aninstance. For example, the image of the form can have been created by ascanner with a first resolution, and the image of the form template canhave been created by a camera with a second resolution, where the firstresolution and second resolution are different. These differentresolutions can cause the size of the scanned image (i.e., the image ofthe form) and the size of the photographic image (i.e., the image of theform template) to differ in the size. For example, a table in thescanned image can be 1000 by 1000 pixels in size, and the image of thecorresponding table in the photographic image can be 2000 by 2000 pixelsin size. In order to facilitate aligning the table of the form with thetable of the form template, one or both of the scanned image and thephotographic image can be scaled so that both tables are at a samescale. For example, the scanned image can be scaled from 2000 by 2000pixels in size to 1000 by 1000 pixels in size, which causes both thescanned image and the photographic image to be at a same scale, which,in this example, is 1000 by 1000 pixels.

Further, when the photo of the form template was taken, the table can beoffset in the photographic image relative to the scanned image of theform. For example, the scanned image may have the table centered in thescanned image, while the photographic image may have the table in acorner of the photographic image. In order to facilitate aligning theimage of the table of the form with the image of the table of the formtemplate, one or both of the scanned image and the photographic imagecan be offset so that both tables are at a same location.

For example, the scanned image can have the center of the table atcoordinate (500, 500) of the scanned image, while the photographic imagecan have the center of the table at coordinate (100, 100) of thephotographic image. In this example, the photographic image can beshifted by 400 pixels in both the X and Y directions, so that the centerof the table of the photographic image is shifted from (100, 100) to(500, 500).

With both tables now having a scale of 1000 by 1000 pixels, and with thecenter of both tables being at (500, 500) in their respective images,the computer system can generate a location aligned version of the thinfeature image (block 925) by applying the scaling and shifting to therotation aligned version of the thin feature image. The location alignedversion of the thin feature image can be represented in a TIFF, JPEG,etc. format, can be represented by data stored in memory, etc. In thisexample, the image of the table of the rotation aligned version of thethin feature image is 1000 by 1000 pixels in size, and the center of thetable is at (500, 500) of the rotation aligned version of the thinfeature image. The computer system can scale and shift the rotationaligned version of the thin feature image using any applicabletechnique. FIG. 12 describes an example of a technique that can be usedto scale and shift the rotation aligned version of the thin featureimage.

At block 930, the computer system refines an alignment of the locationaligned version of the thin feature image. When refining the alignment,the grid lines of the table of the location aligned version of the thisfeature image are more closely aligned with the grid lines of the imageof the form template. The computer system can apply any appropriatetechnique to accomplish the refinement of the alignment. FIG. 13describes an example of a technique that can be used to refine analignment of the location aligned version of the thin feature image withthe image of the form template.

With the grid lines of the table of the location aligned version of thethis feature image now being more closely aligned with the grid lines ofthe image of the form template, a determination can be made that theform is an instance of the form template, and the text in the fields ofthe table of the form can be automatically extracted and captured usingprocesses like those described above.

FIG. 10 is a flow diagram that illustrates a method for generating athin feature image of a form, consistent with various embodiments. Block910 of FIG. 9 can be accomplished using the method of FIG. 10, amongother methods. At block 1005, a computer system selects a pixel of animage of a form, the pixel being referred to as the selected pixel. Atblock 1010, the computer system selects multiple neighboring pixels ofthe selected pixel. A neighboring pixel can be any pixel that has apre-defined location relative to the selected pixel. In an example, aneighboring pixel is a pixel that abuts the selected pixel horizontally,vertically, or diagonally. If one envisions a three by three matrix ofpixels with the center pixel being the selected pixel, for this examplethe neighboring pixels would the remaining eight pixels. In anotherexample, a pixel is a neighboring pixel if it is not the selected pixel,and is within a five by five matrix of pixels where the center pixel isthe selected pixel. In yet another example, a pixel is a neighboringpixel if it is not the selected pixel, and is within a one by 5 matrixof pixels where the center pixel is the selected pixel. When theselected pixel is near to the edge of the image, such that the matrixextends beyond the edge of the image, the matrix size can be reduced sothat it does not extend beyond the edge of the image.

At block 1020, the computer system determines a representative pixel.The representative pixel can be determined in any of various ways, suchas based on the neighboring pixels, based on the neighboring pixels andthe selected pixel, etc. In some embodiments, the representative pixelis determined based on a selected pixel group that includes theneighboring pixels and the selected pixel. For example, therepresentative pixel can be the median pixel of the selected pixelgroup. In this example, the pixels are sorted by value, and the pixel inthe middle of the sorted list of pixels is identified as therepresentative pixel. As another example, the representative pixel canbe the average of the pixels of the selected pixel group. In thisexample, the values of the pixels are summed to generate a dividend. Thedividend is divided by a divisor that is set to the number of pixels inthe selected pixel group. The average of the pixel values is thequotient of the division. The representative pixel can be set to a valueequal to the average of the pixel values.

At block 1025, the computer system replaces the selected pixel with therepresentative pixel, such as in a filtered version of the image. Atblock 1030, the computer system makes a decision whether each pixel ofthe image has been processed. When one or more pixels of the image havenot been processed, block 1005 is executed again on another pixel of theimage of the form. When all pixels of the image have been processed,block 1035 is executed next. At this point, each pixel of the image ofblock 1005 has been replaced with a representative pixel in the filteredversion of the image. At block 1035, the computer system generates afiltered image, such as by storing the filtered version of the image ata storage device, or by storing data in memory that represents thefiltered image.

At block 1040, the computer system subtracts the filtered image from theimage of the form. To subtract a first image from a second image, thevalue of each pixel of the first image is subtracted from the value of acorresponding pixel of the second image. For example, the value of thepixel at location (0, 0) of the first image (the (0, 0) pixel of thefirst image) is subtracted from the value of the (0, 0) pixel of thesecond image, the value of the (0, 1) pixel of the first image issubtracted from the value of the (0, 1) pixel of the second image. Thisprocess is continued until each pixel in the first image has beensubtracted from its corresponding pixel of the second image.

When the pixel value of the first and second images are equal or areclose in value, the result of the subtracting the two pixels is a pixelwith a zero or very small value. However, when the value of the pixel ofthe first image is zero or very small, such as when a thin feature isfiltered out of a pixel of the first image, the result of subtractingthe pixel of the first image from the pixel of the second image is apixel value that equals or is very close in value to the value of thepixel of the second image. When two images are identical or nearlyidentical, the result of subtracting the two images is a blank or nearlyblank image. When two images are nearly identical, except that the firstimage has thin features filtered out, the image that results fromsubtracting the first image from the second image is an image thatincludes the thin features that were filtered out of the first image,but are in the second image. At block 1045, the computer systemgenerates a thin feature image of the form, such as by storing the imagecreated as the result of block 1040 to a TIFF, JPEG, etc. image file, orby storing a representation of the image in a memory of the computersystem.

FIG. 11 is a flow diagram that illustrates a method for generating arotation aligned representation of a table of a form, consistent withvarious embodiments. Block 915 of FIG. 9 can be accomplished using themethod of FIG. 11, among other methods. When an image of a form thatincludes a table is generated, the image may differ from an image of atemplate of the form. For example, the image of the form may have adifferent angular orientation than the image of the form template. Thiscan happen for a variety of reasons, such as due to the form beingrotationally askew when the form is scanned by the scanner to create theimage of the form, due to a camera that is used to generate the image ofthe form not being held at a proper rotational angle relative to theform, etc. As another example, the image may be warped or otherwisedisfigured with respect to the image of the form template or the formtemplate. This can similarly happen for a variety of reasons, such asdue to the paper of the form being bent or otherwise disfigured when theimage of the form is obtained, due to the image capturing device beingoriented at an angle with reference to the plane of the form, etc.

At block 1105, a computer system identifies sub-blocks of a thin featureimage of a form. In order to accelerate processing of the thin featureimage, the thin feature image can be conceptually broken into pieces inorder to reduce the run times, memory image size, etc. when processingthe thin feature image, in order to enable parallel processing of thethin feature image, etc. The sub-blocks can be sized to achieve adesired run time, to achieve a desired granularity or sub-block size, toachieve a desired memory image size, to enable a desired level ofparallel processing, or for any of a number of other reasons. In someembodiments, the number of sub-blocks identified is one.

As an example of identifying sub-blocks, when a thin feature image of aform is 1000 by 1000 pixels (with pixel coordinates ranging from (0, 0)to (999, 999), and the desired size of a sub-block is 100,000 pixels,the thin feature image can be conceptually broken into ten 100 by 1000pixel sub-blocks. A first sub-block, which is comprised of the pixelsfrom coordinate (0, 0) to (99, 999), can be created. A second sub-block,which is comprised of the pixels from coordinate (100, 0) to (199, 999),can be created. This process can continue until a tenth sub-block, whichis comprised of the pixels from coordinate (900,0) to (999,999), iscreated. Separate data structures or other representations of eachsub-block, such as JPEG, TIFF, etc. files, can be created for eachsub-block.

At block 1110, the computer system determines a rotation of grid lies ofa sub-block relative to an X axis or a Y axis. In some embodiments, thecomputer system determines two rotations of grid lines, a first relativeto an X-axis (e.g., that corresponds to rotation of horizontal gridlines relative to the X-axis), and a second relative to a Y-axis (e.g.,that corresponds to rotation of vertical grid lines relative to theY-axis). The X axis can run horizontally and the Y axis vertically withreference to the thin feature image, or with reference to alignment ofpixels of the thin feature image, and the X and Y axes can beperpendicular. It may be instructive to leverage the example of FIG. 8A.FIG. 8A can represent a form where a user enters data in table 800 a ofthe form. Further, a thin feature image of the form of FIG. 8A can besubstantially or nearly identical to the form of FIG. 8A, as the form of8A may have only lines and text, which may not be filtered out in a thinfeature image of the form of FIG. 8A. In a case where the thin featureimage of the form of FIG. 8A is substantially identical to the form ofFIG. 8A, an X axis can run parallel to the horizontal grid lines oftable 800 a, and a Y axis can run parallel to the vertical grid lines oftable 800 a. In such a case, the grid lines of any sub-block would haveno rotation relative to the X and Y axes, as the horizontal grid linesrun parallel to the X axis, and the vertical grid lines run parallel tothe Y axis.

It may be further instructive to leverage the example of FIG. 8C. FIG.8C can similarly represent a thin feature image. The rows of pixels ofthe image represented by FIG. 8C can run horizontally with relative tothe illustration of FIG. 8C, and the X axis can similarly runhorizontally. However, the grid lines of table 800 c of FIG. 8C do notrun parallel to either the X or Y axes of FIG. 8C, but rather run at anangle with reference to the axes.

The rotation of the grid lines can be determined using any of varioustechniques. For example, a Hough transform can be used to detect therotation of the grid lines. A Hough transform can also be used to detectthe grid lines. The sub-block can be Hough transformed with angularsamples relative to the X or Y axes in a pre-determined range, such asfrom −20 degrees to +20 degrees relative to the X axis or the Y axis.The dominant orientations in each sub-block can be determined by summingthe squared Hough-transform bins for each angular sample and selectingthe maximum. The Hough transform can be calculated by rotating eachsub-block by each orientation, and summing vertically or horizontally.

A Hough transform is a feature extracting technique used in imageanalysis, and can be used to identify lines in an image. A line can beexpressed with two variables. For example, in the Cartesian coordinatesystem, a line can be expressed with parameters (m,

) in the form

=mx+

. In the Polar coordinate system, a line can be expressed withparameters (

,θ) in the form

=x cos Θ+

, sine where

is the distance from the origin to the closest point on the straightline, and θ is the angle between the x axis and the line connecting theorigin with that closest point. Graph 1500 of FIG. 15 includes ademonstrative illustration.

It is therefore possible to associate with each line of the image a pair(

,θ). The (

,θ) plane is sometimes referred to as Hough space for the set ofstraight lines in two dimensions.

In general, for each point (x₀,y₀), the family of lines that goesthrough that point can be defined as:

_(θ)=x₀ cos θ+

₀ sin θ, meaning that each pair (

_(θ),Θ) represents each line that passes by (x₀,y₀). If for a given(x₀,y₀) the family of lines that goes through the given point isplotted, a sinusoid is produced. For instance, for x₀=8 and y₀=6, graph1600 of FIG. 16 is produced (in a plane θ-

):

Only points such that

>0 and 0<θ<2π are considered.

The linear Hough transform uses a two-dimensional array, called anaccumulator, to detect the existence of a line described by r=x cos θ+ysin θ. The dimension of the accumulator equals the number of unknownparameters, e.g., two, considering quantized values of r and θ in thepair (r,θ). For each pixel at (x,y) and its neighborhood, the Houghtransform algorithm can determine if there is enough evidence of astraight line at that pixel. If so, it can calculate the parameters(r,θ) of that line, and then look for the accumulator's bin that theparameters fall into, and increment the value of that bin. By findingthe bins with the highest values, such as by looking for local maxima inthe accumulator space, the most likely lines can be extracted, and their(approximate) geometric definitions determined. One way of finding thesepeaks is by applying some form of threshold. Other techniques can beused as well.

A result of the linear Hough transform can be a two-dimensional array(matrix) similar to the accumulator—where one dimension of this matrixis the quantized angle θ and the other dimension is the quantizeddistance r. Each element of the matrix has a value equal to the numberof points or pixels that are positioned on the line represented byquantized parameters (r,θ). So the element with the highest valueindicates the straight line that is most represented in the input image.

At block 1115, the computer system calculates a first transformationthat aligns the grid lines with the X axis or the Y axis. In someembodiments, block 115 includes calculating two alignmenttransformations, a first alignment transformation that aligns horizontallines with the X-axis, and a second alignment transformation that alignsvertical lines with the Y-axis. An alignment transformation can bedetermined based on any of a number of algorithms. Examples of somealgorithms that can be used for an alignment transformation can be foundat http://leptonica.com/rotation.html (an archive of which can be foundathttps://web.archive.org/web/20150318192803/http://leptonica.com/rotation.html).The alignment transformation can be, among others: a rotation bysampling, which chooses the value of each destination pixel to be thatof the source pixel closest to the location the destination pixel camefrom (i.e., before rotation); a rotation by shear, which, depending onthe implementation, is an approximation to rotation by sampling; or arotation by area mapping, which computes the value of each destinationpixel from four source pixels from which it was derived, suitablyweighted by the actual overlap. In some embodiments, an alignmenttransformation is a rotational offset of one or more grid lines from anX-axis or a Y-axis (e.g., horizontal grid lines have a −2% rotationalorientation relative to the X-axis, and vertical grid lines have a −1.5%rotational orientation relative to the Y-axis). In some embodiments, forexample, when the computer system calculates a first rotational offsetfor a vertical line and a second rotational offset for a horizontalline, the computer system can solve for a polynomial warp which mapsthese orientations to vertical and horizontal, using a robust errorfunction to reject outliers. The computer system can apply a costfunction to the detected orientations in each subblock by taking twopoints that are collinear in the detected orientation, and applying thecost function to the difference in warped y-coordinate for thehorizontal orientations and the difference in warped x-coordinate forthe vertical orientations.

At block 1120, the computer system applies the first transformation tothe sub-block, which results in the creation of an intermediate versionor representation of the sub-block where the grid lines aresubstantially aligned with either the X axis or the Y axis. Grid linesthat are aligned with the X axis run horizontally, and grid lines thatare aligned with the Y axis run vertically.

At block 1125, the computer system calculates a second transformationthat linearizes the grid lines. Grid lines may need to be linearizedwhen they are warped or otherwise disfigured with respect to a straightline. As previously discussed, linearizing a line causes the line to runin a straight or substantially straight line, and can involve removingor reducing bends, warps, discontinuities, or other non-straightfeatures of the line. Any of various algorithms can be used to linearizea grid line. For example, a warping model can be used, such as apolynomial warping model of the form u(x,y)=a₀x+a₁y+a₂x{circumflex over( )}2+a₃xy+a₄y{circumflex over ( )}2+ . . . ,v(x,y)=b₀x+b₁y+b₂x{circumflex over ( )}2+b₃xy+b₄y{circumflex over ( )}2+. . . , which maps the input coordinate (x,y) to the output coordinate(u,v).

In some embodiments, based on detection of the grid lines of thesub-block by the Hough transform, a first grid line is identified. For agrid line that runs horizontally, the horizontal grid line shouldideally be linear, and, when the horizontal grid line is not linear, itmay need to be linearized. When a horizontal grid line is linear, theleft and right end points of the horizontal grid line, and all theremaining points of the horizontal grid line, have a same orsubstantially same y-coordinate. For a grid line that runs vertically,the vertical grid line should ideally be linear, and, when the verticalgrid line is linear, the top and bottom end points of the vertical gridline, and all the remaining points of the vertical grid line, shouldhave a same or substantially same x-coordinate. Due to warpage or otherdisfigurement of a grid line, a grid line may not be linear. Forexample, some or even most of the points of a horizontal grid line maynot be at or substantially at a same y-coordinate, or some or even mostof the points of a vertical grid line may not be at or substantially ata same x-coordinate.

To facilitate or accomplish linearizing a grid line, a transformation iscalculated that linearizes the points of the grid line such that, forhorizontal grid lines, the points are at or substantially at a samey-coordinate, and for vertical grid lines, the points are at orsubstantially at a same x-coordinate. The grid lines can be transformedusing any of various algorithms. For example, the grid lines can betransformed based on a warping model, such as the polynomial warpingmodel previously discussed.

Returning to the first grid line, when the first grid line is ahorizontal line and is not linear, the computer system can create areference straight horizontal line that ends at the x-coordinates of theend points of the first grid line. When the first grid line is avertical line and is not linear, the computer system can create areference straight vertical line that ends at the y-coordinates of theend points of the first grid line. A warping model, such as thepreviously discussed polynomial warping model, can be used to linearizethe first grid line. The warping model can be used to minimize adifference between a y-coordinate of the first grid line and ay-coordinate of the reference straight horizontal line when the firstgrid line is a horizontal grid line, and can be used to minimize adifference between an x-coordinate of the first grid line and anx-coordinate of the reference straight vertical line when the first gridline is a vertical grid line. While a standard least squares algorithmcould be used, it has been determined to be sensitive when an outlier ispresent. A cost functions which doesn't penalize large errors assignificantly, such as a robust cost function, can be utilized.

In some embodiments, for example, when the computer system calculates afirst rotational offset for a vertical line and a second rotationaloffset for a horizontal line, the warping model can be used to linearizetwo orientations of lines, such as the vertical and horizontal lines.The warping model can be a polynomial warp which maps these orientationsto vertical and horizontal, using a robust error function to rejectoutliers.

An example of a robust cost function is an M-estimator. Let r_(i) be theresidual of the i^(th) datum, the difference between the i^(th)observation and its fitted value. The standard least-squares methodtries to minimize Σ_(i)r_(i) ², which can be unstable if there areoutliers present in the data. Outlying data can give an effectsufficiently strong in the minimization that the parameters thusestimated are distorted. An M-estimator can reduce the effect ofoutliers by replacing the squared residuals r_(i) ² by another functionof the residuals, yielding

$\begin{matrix}{\min{\sum\limits_{i}^{\;}{\rho\left( r_{i} \right)}}} & \left( {11\text{-}1} \right)\end{matrix}$

where ρ can be a symmetric, positive function with a unique minimum atzero, and can be chosen to be less increasing than square. Instead ofsolving directly this problem, it can be implemented as an iteratedreweighted least-squares one. The following is such an example.

Let p=[p₁, . . . , p_(m)]^(T) be the parameter vector to be estimated.In this example, the M-estimator of p based on the function ρ(r_(i)) isthe vector p which is the solution of the following m equations:

$\begin{matrix}{{{\sum\limits_{i}^{\;}{{\psi\left( r_{i} \right)}\frac{\partial r_{i}}{\partial p_{i}}}} = 0},{{{for}\mspace{14mu} j} = 1},\ldots\mspace{14mu},m,} & \left( {11\text{-}2} \right)\end{matrix}$

where the derivative ψ(x)=dρ(x)/dx is called the influence function.

If a weight function is defined as follows:

$\begin{matrix}{{w(x)} = \frac{\psi(x)}{x}} & \left( {11\text{-}3} \right)\end{matrix}$

then Equation (11-2) becomes

$\begin{matrix}{{{\sum\limits_{i}^{\;}{{w\left( r_{i} \right)}r_{i}\frac{\partial r_{i}}{\partial p_{i}}}} = 0},{{{for}\mspace{14mu} j} = 1},\ldots\mspace{14mu},m} & \left( {11\text{-}4} \right)\end{matrix}$

This is the system of equations that can be obtained if the followingiterated reweighted least-squares problem is solved

$\begin{matrix}{\min{\sum\limits_{i}^{\;}{{w\left( r_{i}^{({k - 1})} \right)}r_{i}^{2}}}} & \left( {11\text{-}5} \right)\end{matrix}$

where the superscript (k) indicates the iteration number. The weightw(r_(i) ^((k-1))) can be recomputed after each iteration in order to beused in the next iteration.

The influence function ψ(x) measures the influence of a datum on thevalue of the parameter estimate. For example, for the least-squares withρ(x)=x²/2, the influence function is ψ(x)=x, that is, the influence of adatum on the estimate increases linearly with the size of its error,which confirms the non-robustness of the least-squares estimate. In someembodiments, a robust estimator can prevent the influence of any singleobservation (datum) from causing a significant offset. There are severalconstraints that some embodiments of a robust M-estimator meets:

The first is to have a bounded influence function.

The second is that the robust estimator be unique. This implies that theobjective function of parameter vector p to be minimized should have aunique minimum. In some embodiments, this can require that theindividual ρ-function is convex in variable p. This can be necessary inthese embodiments because only requiring a ρ-function to have a uniqueminimum may not be sufficient. This can be the case with maxima whenconsidering mixture distribution; the sum of unimodal probabilitydistributions can be multi-modal. The convexity constraint is equivalentto imposing that

$\frac{\partial^{2}{\rho\left( . \right)}}{\partial p^{2}}$is non-negative definite.

The third one is a practical requirement. Whenever

$\frac{\partial^{2}{\rho\left( . \right)}}{\partial p^{2}}$is singular, the objective preferably has a gradient,

$\frac{\partial{\rho\left( . \right)}}{\partial p} \neq 0.$This can avoid having to search through the complete parameter space.

Based on the detection of the grid lines of the sub-block by the Houghtransform, grid lines are identified, and a warping algorithm is appliedto the grid lines. In some embodiments, the algorithm is additionallyapplied to non-grid lines.

At block 1130, the computer system applies the second transformation tothe sub-block, or to the intermediate version or representation of thesub-block that was created at block 1120, which results in in thecreation of a version or representation of the sub-block where the gridlines are substantially linear. At block 1135, a determination is madewhether each of the sub-blocks have been processed. If no, block 1110 isexecuted for one of the unprocessed sub-blocks of the thin featureimage. If all the sub-blocks have been processed, then block 1140 isnext executed. At block 1140, the computer system generates a rotationaligned representation of the thin feature image, such as by storing theversion or representation of the sub-block created at block 1130 to aTIFF, JPEG, etc. image file, or by storing the version or representationof the sub-block at a memory of the computer system. In someembodiments, rather than applying the transformations of blocks 1120 and1130 at the sub-block level, the information determines at blocks 1115and 1125 are used to determine a global transformation, and the globaltransformation is applied to all or a portion of the thin feature imageof the form.

FIG. 12 is a flow diagram that illustrates a method for scaling andshifting a first image of a form that includes a table, consistent withvarious embodiments. Block 920 of FIG. 9 can be accomplished using themethod of FIG. 12, among other methods. The first image of the form canbe the rotation aligned version of the thin feature image that wascreated at block 915 of FIG. 9, and the first image can be scaled andshifted to align with a second image, such as the template of the form.At block 1205, a computer system or a user identify a range of scalesover which to scale a first image, such as from a 50% scale to a 150%scale. With knowledge of the various ways that images of forms arecreated, the user can define maximum and minimum scales that are likelyto happen when images are created, and the user can identify the rangeof scales based on these maximum and minimum scales. As the computersystem gathers data or other statistics as it determines scales offorms, the computer system can refine the range of scales that arelikely to happen when images are created based on this historic data orstatistics. Based on this historic scale data or statistics, thecomputer system can identify, or refine, the range of scales over whichto scale a first image.

At block 1210, a computer system or a user identify a range of X or Yoffsets over which to shift a first image. With knowledge of the variousways that images of forms are created, the user can define a maximumoffset that is likely to happen when an image of a form is created, andthe user can identify a range of offsets based on this maximum offset.As the computer gathers data or other statistics at it determinesoffsets of forms, the computer system can refine the range of offsetsthat are likely to happen when images are created based on this historicoffset data or statistics. Based on this historic offset data orstatistics, the computer system can identify, or refine, the range ofoffsets over which to shift a first image.

At block 1215, the computer system scales the first image in the X or Ydimension by a scale amount. The computer system, at blocks 1215 through1235, iterates though various scale and shift values in an attempt todetermine a scale and a shift value that optimizes an alignment of thefirst image with a second image. In some embodiments, the scaling andshifting is done separately for the x and y dimensions. For example, ina first set of iterations at blocks 1215 through 1235, the computersystem scales the first image in the X dimension (block 1215), shiftsthe scaled first image relative to the second image (block 1220), andcomputes an alignment score (block 1225). The computer system can, in asecond set of iterations at blocks 1215 through 1235, scale the firstimage in the Y dimension (block 1215), shift the scaled first imagerelative to the second image (block 1220), and computes an alignmentscore (block 1225).

The range of scales can run, for example, from a minimum scale, such as80%, to a maximum scale, such as 120%. The range of X or Y offsets canrun, for example, from a first value to a second value. For example, thefirst value can be minus 200 pixels in the X and Y dimensions, and thesecond value can be plus 200 pixels in the X and Y dimensions. Thecomputer system can iterate over this range. For example, the computersystem can scale the first image by 80% (block 1215), and can iterateover various shifts from (−200, −200) pixels to (200, 200) pixels, suchas in 5 pixel increments (block 1220).

As each iteration reaches block 1225, the computer system computes analignment score. Computing an alignment score can include, can result,or can be derived from, calculating a cross-correlation of the scaledshifted first image with the second image, and the cross-correlation canbe used in a determination of an alignment score. The cross-correlationcan be calculated in any of various ways. For example, the value of apixel of the scaled shifted first image can be multiplied by the valueof a pixel of the second image at a same location when the scaledshifted first image and the second image are overlaid. When a grid lineof the scaled shifted first image aligns with a grid line of the secondimage, a dot product of pixels of the two images will show a peak wherethe pixels of the grid lines align. When the two grid lines do notalign, a dot product of pixels of the two images will show smallervalues at the locations of the two grid lines. As the first image isscaled and shifted, the values of the dot products can be monitored, andpeaks in the dot products can identify scales or shifts that maximizealignment between the grid lines of the shifted scaled first image andthe second image. In another example, the cross-correlation iscalculated at each iteration based on a fast Fourier transform (FFT).

After each iteration over the range of X or Y offsets, at block 1230, adetermination is made whether the first image has been shifted over therange of X or Y offsets. If no, block 1220 is executed next at the nextX or Y offset. If yes, at block 1235, a determination is made whetherthe first image has been scaled over the range of scales. If no, block1215 is executed next at the next scale amount. If yes, then block 1240is executed next. At block 1240, the computer system determines a scalevalue and a shift value that optimizes a cross-correlation of the firstimage and the second image. The scale value, which may be a differentscale in the X and Y dimensions, or may be a same value in bothdimensions, stretches or shrinks the first image to cause a table in thefirst image to be substantially the same size as a table in the secondimage. The shift value, which is an offset of the first image in the Xand Y dimensions, causes a table in the first image to substantiallyalign with a table in the second image.

The orders of the blocks of FIG. 12, as well as the other figures, canbe varied. For example, in some embodiments, the shifting of block 1220occurs before the scaling of block 1214 for each iteration. Further, insome embodiments, the scaling and shifting are done as one operation.

FIG. 13A is a flow diagram that illustrates a first method of refiningan alignment of an image of a table of a form with an image of a tableof a template of the form, consistent with various embodiments. Block930 of FIG. 9 can be accomplished using the method of FIG. 13A or 13B,among other methods. After the thin feature image of the form islocation aligned to the image of the form template, an alignment of thetable of the thin feature image with the table of the image of the formtemplate can be further refined. As a result of the thin feature imagebeing location aligned, when the thin feature image of the form isoverlaid over the image of the template of the form, the grid lines ofthe table of the thin feature image line up pretty well with the gridlines of the table of the form template image. The two images have beenrotation aligned, have been scaled to a substantially same scale, havebeen positionally aligned, and the grid lines have been straightened. Asa result, the grid lines of both tables align pretty well. Now that thethin feature image has been location aligned to the form template, thealignment can be refined based on any of various algorithms, such as aone or two dimensional non-linear alignment. Further, because of theachieved close alignment of the two tables, methods of refining thealignment that are too computationally expensive when the two tables arebadly aligned, but are computationally tolerable when the two tables arewell aligned, can be used.

At block 1305, a computer system identifies sub-blocks of a first imagethat is aligned with a second image. The first image can be the locationaligned version of the thin feature image that was generated at block925 of FIG. 9, and the second image can be the template of the form thatwas received at block 905 of FIG. 9. This block can be substantiallysimilar to block 1105 of FIG. 11, except that the sub-blocks aredetermined for the first image.

At block 1310, the computer system selects a sub-block of the firstimage and a corresponding sub-block of the second image. Thecorresponding sub-block of the second image can be, e.g., a sub-blockcomprised of a set of pixels that fall within a boundary of the selectedsub-block when the first image is overlaid over the second image. Asanother example, the corresponding sub-block can be a sub-blockcomprised of a set of pixels that are at a same location as the pixelsof the sub-block of the first image when the first image is overlaidover the second image.

At block 1315, the computer system displaces the sub-block of the firstimage by a first displacement. In some embodiments, the firstdisplacement is a displacement in one dimension, such as a displacementin the X dimension, or a displacement in the Y dimension (but not adisplacement in both dimensions). In other embodiments, the firstdisplacement is a displacement in two dimensions. For example, when thefirst displacement in a displacement in one dimension, the computersystem can displace the sub-block by one pixel in the positive xdirection, or by two pixels in the negative x direction, etc. Forexample, when the first displacement in a displacement in twodimensions, the computer system can displace the sub-block by one pixelin the positive x direction and one pixel in the positive y direction,or by two pixels in the negative x direction and one pixel in thepositive y direction, or by three pixels in the positive x direction andzero pixels in the y direction, etc.

At block 1320, the computer system calculates a cross-correlationbetween the displaced sub-block and the corresponding sub-block. In someembodiments, a correlation is a dot product after a mean is subtractedfrom each signal, and the cross-correlation is the correlation at eachof multiple displacements. In a first example a grid line of the firstimage is one grid thick, and is misaligned with a corresponding gridline of the second image by one pixel. When the sub-block containing thegrid line of the first image is displaced by one pixel such that thegrid line of the first image and the grid line of the second imagealign, the dot product of the two sub-blocks will produce a spike wherethe two grid lines align. When the sub-block containing the grid line ofthe first image is displaced by two pixels, the two grid lines will notalign, and the cross-correlation of the two sub-blocks will not producea spike, as the two grid lines do not align. The sub-block of the firstimage can be displaced within a predetermined one dimensional or twodimensional range, and a dot product can be calculated for eachdisplacement.

In some embodiments, a score is calculated for each of multipledisplacements of each sub-block. The computer system then solves aMarkov Random Field (MRF) that maximizes an alignment of individualblocks and smoothness with respect to the displacement of neighboringblocks. An OpenGM library, such as the one located athttp://hci.iwr.uni-heidelberg.de/opengm2/ (an archive copy of which canbe found athttps://web.archive.org/web/20151024082917/http://hci.iwr.uni-heidelberg.de/opengm2/)can be leveraged to calculate the MRF. Calculating a MRF is known tothose in the art. For example, seehttp://www.cs.cornell.edu/˜rdz/Papers/SZSVKATR.pdf (an archive copy ofwhich is available athttps://web.archive.org/web/20151113020857/http://www.cs.cornell.edu/˜rdz/Papers/SZSVKATR.pdf).

In a second example, the sub-block of the first image can be displacedwithin a predetermined one dimensional or two dimensional range, and acorrelation can be calculated for each displacement. The negative of thecorrelation as can be used as unary terms in a four-connected grid MRF,with a label for each discrete one or two dimensional displacement. Forpairwise terms between labels, the Euclidian distance betweencorresponding displacement vectors can be used, raised to a power, suchas the power three. The pairwise cost can weakly penalize many smalldifferences in displacement, and heavily penalize a single largedisplacement. The MRF model is similar to those used for stereo matchingin image processing, such as for computer vision. However, in thisapplication, a search over two dimensional displacements rather than onedimensional displacements can be done, and the pairwise term is bettersuited for finding an everywhere-smooth mapping, rather than thepiecewise-constant or piecewise-smooth labels favored by stereo priors.

One difference between the method of this figure and with stereomatching in computer vision is worth noting. In image processing, adiscontinuity can happen when, for example, a person is in theforeground, and the background is a building 100 yards behind theperson. When processing the image of the person's face, for example, adiscontinuity from the edge of the face to the building 100 yards behindthe person is to be expected for computer vision. So a big differencebetween two neighboring pixels is to be expected and is fine. However,for the method of this figure, a big difference is worse than a smalldifference. This is because the grid lines have already been fairlyclosely aligned, so there should be no major displacements. This goes tothe reason for the particular pairwise cost that is used in the methodof this figure, and why the pairwise cost weakly penalizes many smalldifferences in displacement, and heavily penalizes a single largedisplacement.

The MRF can be solved with the OpenGM library discussed above, using theAlphaExpansionFusion solver, giving a displacement for each sub-block.The displacement field can be upsampled to the full resolution of thetemplate using Bicubic interpolation. Seehttps://en.wikipedia.org/wiki/Bicubic_interpolation (an archive copy ofwhich is available athttps://web.archive.org/web/20150925180553/https://en.wikipedia.org/wiki/Bicubic_interpolation).

At block 1325, the computer system refines an alignment of the firstimage with the second image based on the calculated cross-correlation.As discussed at block 1320, the sub-block of the first image can bedisplaced within a predetermined one dimensional or two dimensionalrange, and a cross-correlation can be calculated for each displacement.When calculating the cross-correlation involves calculating a dotproduct, as in the first example of block 1320, a spike or maximum inthe dot product can indicate that the particular displacement of thatcalculation improves an alignment of a grid line in the first image anda grid line in the second image. The first image, or a portion of pixelsof the first image, can be displaced by the particular displacementamount, or an amount determined based on the particular displacementamount, in order to refine the alignment of the first image and thesecond image. The portion of the pixels of the first image can be thepixels of the sub-block of the first image.

When calculating a cross-correlation involves calculating a MRF, as inthe second example of block 1320, a measure of cross-correlation can bebased on the solutions to the MRFs at the various displacements of thesub-block of the first image. The values of the solutions to the MRFscan indicate an improved alignment of a grid line of the first image anda grid line of the second image. Just as above, the first image, or aportion of pixels of the first image, can be displaced by the particulardisplacement amount, or an amount determined based on the particulardisplacement amount, in order to refine the alignment of the first imageand the second image. For example, when the cross-correlation indicatesan improved or optimal alignment between the sub-block of the firstimage and the corresponding sub-block of the second image, the firstimage or the portion of the first image can be displaced by theparticular displacement amount in order to refine the alignment of thetwo images.

At block 1330, a determination is made whether each of the sub-blockshave been processed. If no, block 1310 is executed for one of theunprocessed sub-blocks of the first image. If all the sub-blocks havebeen processed, then block 1335 is next executed. At block 1335, thecomputer system generates a refined alignment representation of thefirst image, such as by storing the refined alignment representation toa TIFF, JPEG, etc. image file, or by storing the refined alignmentrepresentation at a memory of the computer system.

In some embodiments, the method of FIG. 13 is repeated multiple times,each time with a different algorithm. For example, the method of FIG. 13can be executed a first time based on a one dimensional displacement ofsub-blocks. After this initial refinement is achieved, the method ofFIG. 13 can be executed a second time based on a two dimensionaldisplacement of sub-blocks. This can be useful, as the initial onedimensional refinement is less compute intensive than the second twodimensional refinement, and starting the two dimensional refinement withthe refined alignment result of the one dimensional alignment reducesthe computational expense of running the more computationally expensivetwo dimensional displacement.

FIG. 13B is a flow diagram that illustrates a second method of refiningan alignment of an image of a table of a form with an image of a tableof a template of the form, consistent with various embodiments. Blocks1355-1360 are, respectively, similar to blocks 1305-1310. At block 1365,the computer system displaces the sub-block of the first image bymultiple displacements. The displacements can be, e.g., all possibledisplacements over a displacement range with each displacement limitedby a minimum stepping size, a subset of the possible displacements overthe range (e.g., multiples of five times the stepping size, multiples often times the stepping size, etc.). At block 1370, the computer systemcalculates a correlation of the sub-block at each displacement. At block1375, a determination is made whether each of the sub-blocks of thefirst image has been processed. If no, block 1360 is executed for one ofthe unprocessed sub-blocks of the first image.

At block 1380, the computer system generates a cross-correlation basedon the correlation of block 1370. The cross-correlation can be based oncorrelations of a subset of all the blocks processed at block 1370, andthe subset can be any or all of the blocks processed at block 1370. Atblock 1385, the computer system optimizes an MRF based on thecorrelations, such as based on the cross-correlation of block 1380, asubset of the correlations of block 1370, etc. The MRF can be based onthe sum of a subset of the correlations of block 1370. For example, theMRF can measure the sums of the subset of the correlations of block1370. The MRF can also, or instead, be based on the cross-correlation ofblock 1380. The MRF can further be based on smoothness of displacementsof neighboring patches. At block 1390, the computer system up-samplesthe per-sub-block displacement field and per-pixel displacement field towarp the image of the table.

FIG. 14 is a high-level block diagram showing an example of a processingdevice 1400 that can represent a system to run any of themethods/algorithms described above, consistent with various embodiments.A system may include two or more processing devices such as representedin FIG. 14, which may be coupled to each other via a network or multiplenetworks. A network can be referred to as a communication network.

In the illustrated embodiment, the processing device 1400 includes oneor more processors 1410, memory 1411, a communication device 1412, andone or more input/output (I/O) devices 1413, all coupled to each otherthrough an interconnect 1414. The interconnect 1414 may be or includeone or more conductive traces, buses, point-to-point connections,controllers, adapters and/or other conventional connection devices. Eachprocessor 1410 may be or include, for example, one or moregeneral-purpose programmable microprocessors or microprocessor cores,microcontrollers, application specific integrated circuits (ASICs),programmable gate arrays, or the like, or a combination of such devices.The processor(s) 1410 control the overall operation of the processingdevice 1400. Memory 1411 may be or include one or more physical storagedevices, which may be in the form of random access memory (RAM),read-only memory (ROM) (which may be erasable and programmable), flashmemory, miniature hard disk drive, or other suitable type of storagedevice, or a combination of such devices. Memory 1411 may store data andinstructions that configure the processor(s) 1410 to execute operationsin accordance with the techniques described above. The communicationdevice 1412 may be or include, for example, an Ethernet adapter, cablemodem, Wi-Fi adapter, cellular transceiver, Bluetooth transceiver, orthe like, or a combination thereof. Depending on the specific nature andpurpose of the processing device 1400, the I/O devices 1413 can includedevices such as a display (which may be a touch screen display), audiospeaker, keyboard, mouse or other pointing device, microphone, camera,etc.

Unless contrary to physical possibility, it is envisioned that (i) themethods/steps described above may be performed in any sequence and/or inany combination, and that (ii) the components of respective embodimentsmay be combined in any manner.

The techniques introduced above can be implemented by programmablecircuitry programmed/configured by software and/or firmware, or entirelyby special-purpose circuitry, or by a combination of such forms. Suchspecial-purpose circuitry (if any) can be in the form of, for example,one or more application-specific integrated circuits (ASICs),programmable logic devices (PLDs), field-programmable gate arrays(FPGAs), etc.

Software or firmware to implement the techniques introduced here may bestored on a machine-readable storage medium and may be executed by oneor more general-purpose or special-purpose programmable microprocessors.A “machine-readable medium”, as the term is used herein, includes anymechanism that can store information in a form accessible by a machine(a machine may be, for example, a computer, network device, cellularphone, personal digital assistant (PDA), manufacturing tool, any devicewith one or more processors, etc.). For example, a machine-accessiblemedium includes recordable/non-recordable media (e.g., read-only memory(ROM); random access memory (RAM); magnetic disk storage media; opticalstorage media; flash memory devices; etc.), etc.

Note that any and all of the embodiments described above can be combinedwith each other, except to the extent that it may be stated otherwiseabove or to the extent that any such embodiments might be mutuallyexclusive in function and/or structure.

Although the present invention has been described with reference tospecific exemplary embodiments, it will be recognized that the inventionis not limited to the embodiments described, but can be practiced withmodification and alteration within the spirit and scope of the appendedclaims. Accordingly, the specification and drawings are to be regardedin an illustrative sense rather than a restrictive sense.

The invention claimed is:
 1. A method for aligning grid lines of animage of a table of a filled-out paper form with grid lines of an imageof a table of a template of the filled-out paper form, the methodcomprising: generating, by a computer system, a thin feature image of afilled-out paper form by: applying a median filter to an image of afilled-out paper form to generate a filtered image, wherein the medianfilter is configured to eliminate image features below a pre-definedthickness, wherein the pre-defined thickness is based on a thickness ofa target grid line; subtracting the filtered image of the filled-outpaper form from the image of the filled-out paper form to generate thethin feature image, wherein the thin feature image includes grid linesof a table of the filled-out paper form; straightening, by the computersystem, the grid lines of the table by: aligning the grid lines of thetable with an X axis or a Y axis, the X axis being a horizontal axis andthe Y axis being a vertical axis, by: for each sub-block of a pluralityof sub-blocks of the thin feature image: calculating a plurality ofHough transforms for said each sub-block, each of the plurality of Houghtransforms being calculated based on a different rotation of said eachsub-block relative to the X axis or the Y axis, determining a dominantrotation relative to the X axis or the Y axis for said each sub-blockbased on the plurality of Hough transforms, calculating a transformationfor said each sub-block based on the dominant rotation, wherein thetransformation aligns lines in said each sub-block with the X axis orthe Y axis, and applying the transformation to generate a rotationaligned version of said each sub-block, and generating the rotationaligned version of the thin feature image based on the rotation alignedversions of the plurality of sub-blocks; and scaling and shifting, bythe computer system, the rotation aligned version of the thin featureimage by: determining a scale translation and a shift translation thatmaximizes a correlation between the grid lines of the table of therotation aligned version of the thin feature image, and template gridlines of an image of a template table of a form template, and applyingthe scale translation and the shift translation to the rotation alignedversion of the thin feature image to generate a location aligned versionof the thin feature image, wherein an alignment of the grid lines of thetable of the location aligned version of the thin feature image with thetemplate grid lines of the image of the template table facilitates adetermination that the table and the template table are aligned.
 2. Themethod of claim 1, further comprising: performing a one dimensionalrefinement of the alignment by: for each pixel of the location alignedversion of the thin feature image: displacing said each pixel by a firstdisplacement in either a first direction that is parallel with the Xaxis or a second direction that is parallel with the Y axis, thedisplacing of the pixel being to a displaced location, selecting a pixelof the form template that is at a same position as the displacedlocation, selecting a first value associated with said each pixel,selecting a second value associated with the pixel of the form template,calculating a third value by performing a mathematical operation thatincludes the first value and the second value, and refining thealignment based on the third value.
 3. The method of claim 1, furthercomprising: performing a two dimensional refinement of the alignment by:for each pixel of the location aligned version of the thin featureimage: displacing said each pixel by a first displacement in a firstdirection that is parallel with the X axis, and a second displacement ina second direction that is parallel with the Y axis, the displacing ofthe pixel being to a displaced location selecting a pixel of the formtemplate that is at a same position as the displaced location, selectinga first value associated with said each pixel, selecting a second valueassociated with the pixel of the form template, calculating a thirdvalue by performing a mathematical operation that includes the firstvalue and the second value, and refining the alignment based on thethird value.
 4. The method of claim 1, wherein the calculating theplurality of Hough transforms for said each sub-block includescalculating a squared Hough transform bin for each of the plurality ofHough transforms, and wherein the dominant rotation is determined basedon a summing of the squared Hough transform bin for said each of theplurality of Hough transforms.
 5. The method of claim 1, wherein thestraightening the grid lines of the table further includes calculatingand applying a second transformation that linearizes the grid lines ofthe table.